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ABSTRACT 

The New Horizons project was a workplace literacy 
partnership during which 454 employees (53%) of Georgetown Steel 
attended classes provided by Horry-Georgetown Technical College in 
Conway, South Carol.' r.a. Of the 454 participants, 294 were white, 159 
were black, 71 were female, 383 were male, 133 had been with the 
company for 5 years or less, and 227 for 16 or more years. The 
participants, who had a mean age of 40.6 years, received a total of 
14,232 hours of instruction in math, reading, writing, problem 
solving, rnd time management classes averaging 6~8 students per 
class. A comprehensive needs assessment was conducted to identify 
employees requiring basic skills instruction. Individualized 
Education Plans were developed for each participant. A comprehensive 
evaluation of trainee outcomes that included pre- and posttesting and 
interviews with participants and supervisors established that 
participants averaged 227., 397., and 147. gains in reading, math, and 
writing skills, respectively. Also included is a third-party 
evaluation report confirming the significant skill gains achieved by 
course participants. Appended to the third-party evaluation are the 
participant and supervisor survey forms, an analysis matrix of 
project objectives, and 22 charts detailing participant survey and 
pre- and posttest data. (MN) 
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Participant Statistics 
June 1,1992 -April 30,1994 



454 Employees attended classes provided by the New Horizons 
Project (This represents 53% of the entire Georgetown Steel 
employee population). 

Mean Age of Participants = 40.6 years 

Female Participants = 71 Male Participant = 383 

Race/Ethnicity = White 294 

Black 159 
Hispanic 0 
Native American 0 
Asian 1 

Contact Hours Provided = 14,232 

Number of Years with the Company = 0-5 133 

6-10 44 

11-15 50 

16 or More 227 
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Objectives, Accomplishments and Observations 

Objective #1 

To identify and select instructors who have a successful background in Adult 
Education and job-related workplace training. To conduct training and orientation 
sessions for all instructors, which will: review effective instructional techniques, 
adult learning theories, program goals and objectives, course goals and 
objectives, Individualized Educational Plans and program policies and 
procedures. To conduct, throughout the project, ongoing assessments and 
evaluations of all instructors. 

Accomplishments: 

All aspect of Objective #1 were met. Although, it was difficult to get all the 
qualified staff on board in accordance to the projected timeline. This was largely 
due to the grant being awarded a month before the College was notified. 

Observations: 

It is difficult to find instructors who have both experience with teaching adults and 
are capable of developing and delivering customized, job-specific instructional 
materials. A crucial characteristic of a successful instructor in a workplace setting 
is the ability and willingness of that instructor to be flexible and creative. 
Traditional instructors who can operate only within the confines of how and what 
they've always taught, even in Adult Education, have a difficult time relating to the 
varying skill levels of their students as well as the necessity of teaching more than 
one subject area. 

Objective #2 

To conduct and complete comprehensive needs assessments to verify the 
workers who require basic skills training and to verify the specific job related 
basic skills required for each participant to most effectively perform his/her job 
and become more productive. To develop competencies (based on the needs 
assessments) for each identified skill. 



Accomplishments: 

Ail aspects of Objective #2 were met on schedule. 
Observations: 

A taxonomy was used to help guide us through the Needs Assessment Process. 
This helped us to ascertain what Math, Reading and Writing Skills were needed 
in the mill at Georgetown Steel Corporation. The following are examples of 
questions that we asked: 

Examples of Reading Questions: 

In you Department do you... 

1. Read Signs (such as Caution, Flammable, etc.)? 

2. Read Forms (such as Work Orders, Job Orders, Purchase Orders, Sales 
Slips etc.)? 

3. Read Manuals and Reference books? 
Examples of Math Questions: 

In your Department do you... 

1. Read Whole Numbers? 

2. Write Whole Numbers given their work names? 

3. Arrange Whole Numbers in order? 

Numerous visits were made to different departments within the Steel Mill. At 
e^ch department, we went through the Taxonomy and asked how they used 
Math, Reading and Writing Skills, to give us examples of how those skills were 
used and copies of any frequently used forms for which those particular skills 
were needed. Not only did the Needs Assessment provide us with valuable 
information on basic skills that are necessary in the Steel Mill, it gave the 
employees the opportunity to be involved in the development of the New Horizons 
Project. This led to increased interest on the part of the employees to participate 
in the resulting New Horizons classes. 

We found that it was crucial to have someone on the Needs Assessment Team 
who knows the plant and its processes. This person provided a crucial link by 
interpreting for both the Needs Assessment Team (academic questions made 
relevant to the mill workers) and the employees in the mill(providing mill answers 
that were relevant to the academic questions). Even taking precautions to make 
sure everyone is "speaking the same language" during the Needs Assessment 
Process, misinterpretations are still a potential problem. For example, during the 
Needs Assessment, we were told by numerous departments that their employees 
needed to read and understand whole numbers. As educators, we interpreted 
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this to mean that place value was an important concept that needed to be 
mastered. Later, we discovered that whole numbers in the mill were treated 
more as identifiers like a Social Security Number or a Drivers License Number. 

Objective #3 

To write measurable performance objectives. To develop course measurable 
objectives. 

Accomplishments: 

All aspects of Objective #3 were met. 
Objective #4 

To develop course objectives and course outlines. To design course content to 
reflect the competencies and skills taught. To develop job related curriculum 
materials, based on the Literacy Task Analysis, that enhance adult learning and 
which are designed to meet the stated competencies, and performance objectives. 

Accomplishments: 

All aspect of Objective #4 were met. However, the time constraints of the grant 
made it extremely difficult to create, adapt and finalize curriculum materials for 
five subjects. 

Observations: 

Materials and information we collected during the Needs Assessment provided 
the framework for the development of the job related curriculum for Math, 
Reading, Writing, Problem Solving and Time Management Classes. 

Overall, having instructors who had little experience with writing curriculum 
proved to be a challenge. The development of curriculum would likely have been 
a smoother process had there been a curriculum specialists who relied on the 
instructors only to provide feedback about the usability of the curriculum in the 
classroom. 

Objective #5 

To develop a written Individualized Educational Plan (IEP) for each participant. 
The lEPs will be negotiated and developed by the participants and the instructors. 

Accomplishments: 

All aspects of Objective #5 were met. 
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Observations: 



lEPs were used for the Math, Reading, and Writing Classes. They were a useful 
tool in guiding the employee through these classes by specifying which course 
objectives they would work on. These course objectives were identified based on 
what each student had missed on his/her pre-test. 

Objective #6 

To conduct basic skills training for at least 484 Georgetown Steel Employees 
during the project period. To offer classes and instruction in: Math, Reading, 
Writing, Problem Solving and Decision Making, and Time Management. To offer 
a program option of one-on-one tutoring for those with no, or extremely low 
reading skills. To provide high school completion/GED preparation classes and 
training to all who desire, or need, to reach this educational level. To provide 
class schedules and class sizes which will meet the needs of all participants and 
enhance learning. To provide an opportunity for every participant to achieve 
his/her basis skills. 

Accomplishments: 

All aspects of Objective #6 were met. 
Observations: 

Instruction was provided for 454 employees of Georgetown Steel. The main 
factors contributing to the project serving 30 fewer employees than projected 
were: 1). Classes had an average of 6 - 8 students as opposed to the 12 - 15 
projected. This was due to physical constraints of the dassroom and tne fact that 
the instructors could offer more one-on-one type instruction when the class size 
was smaller. 2). With the smaller class size and classes offered on an open- 
ended basis, there was often a long wait before there was available space for a 
new student to start a class. All things considered, New Horizons did provide 
classes for more than half of the employees of Georgetown Steel during the 
restricted time frame of the Workplace Literacy Grant Funding. 

Pre-tests for Math, Reading, and Writing were given to those students who 
scored roughly at a 4th grade level and higher on the Bader Reading Inventory. 
Students who scored below the 4th grade level were placed in a class for 
emergent readers. This class was conducted in a small group setting. 

At the beginning of the project, all of the Math and Reading students were placed 
in a class and worked on whatever objectives they had missed on their pre-test. 
We saw a trend of very high pre-test scoros and equal or lower post-test scores. 
Consequently, we decided that anyone scoring 85% or higher would be 



considered passing and would not take part in the classes (The exception was 
that all of the reading students did the vocabulary and dictionary skills portion of 
the curriculum). 

In retrospect, those of us who were involved with the grant felt that more closed 
end, modular type training would have worked better within the environment of 
the mill. That way, students could begin and end classes at a prescribed rate 
which would tend to interfere less with work schedules and production. This 
would have been particularly relevant to the Math and Reading Classes where a 
student could have felt more of a sense of accomplishment in a shorter period of 
time. 

Objective #7 

To conduct a comprehensive evaluation of trainee outcomes as well as an 
evaluation of the training program. 

Accomplishments: 

All aspects of Objective #7 were met. 
Observations: 

Trainee evaluation was done through pre and post testing as well as through 
interviews with participants and their supervisors. Pre to post-test gains in 
Reading , Math and Writing were significant at 22%, 39% and 14% respectively. 
The customized pre and post-tests were a highlight of the program. Not only did 
they show significant increases in skill, they were found, through statistical 
review, to be valid 2nd reliable instruments. 

Survey responses from the employees reflected higher achievement of goals with 
regard to increases in communication and production than shown by the 
responses of the supervisors. However, both showed that positive improvement 
had been made in both areas. 

For additional information, refer to the New Horizons External Evaluation Impact 
Study by our third party evaluator, Dr. John Gretes, 

Objective #8 

To develop and implement a multifaceted media campaign to promote the project 
and inform the community about the benefits of the program. The Project 
Director will generate a quarterly report which will be submitted to key project 
personnel. Appropriate information included in the report will be made available 
for press releases and public service announcements. Upon completion of the 
project, the Director will: 1). submit a Final Report to the United States 



Department of Education; 2). hold a news conference summarizing the project 
and its benefits; 3). submit at least three articles to three journals with national 
circulations; 4). hold a conference open to the public, other agencies, 
corporations and institutions, and 5). disseminate a summary of the project and 
copies of published articles to the South Carolina Work Force Initiative Program 
Specialists. 

Accomplishments: 

All aspects of Objective #8 have been met except for writing and submitting 
articles. It was felt that more effective articles could be written once the final 
evaluation for the project was completed. 

Observations: 

Dissemination was a large part of the New Horizons project. A video was 
produced and successfully used to both recruit students and give visitors a 
comprehensive overview of the training project. Presentations about New 
Horizons were made at a Workforce Education Conferences in Minnesota and 
Wisconsin, a South Carolina Developmental Education Conference and a 
Conference for Community College Trustees in Toronto, Canada. In addition, 
New Horizons was visited by several groups from different educational institutions 
and various industry. These groups came from South Carolina, Florida, Georgia, 
Connecticut, North Carolina, Pennsylvania and South Africa. 



Changes in Key Personnel: 

The only change made in Key Personnel was the replacement of our first project 
evaluator who became ill and was unable to fulfill his commitment. 
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INTRODUCTION 



The project impact study is based on the External Evaluation Plan (revised November 26, 
1993) The first seven pages of this document provide an overview of that plan. The 
remaining sections of this document provide the Project Impact Study results. The Impact 
Study results address each of the four areas defined by the External Evaluation Plan. These 
areas include, (1) Reaction, "Reaction" involves the instructor and participant reactions to 
aspects of the program. It answers the question, "How well did participants like the 
program 0 ", (2) Learning, "Learning, involves analysis of gains made by participants in 
knowledge and skills taught in the program courses.", (3) Behaviors, "Behavior, involves 
changes in participant behavior.", (4) Results, "Results, involve such factors as reduced 
turnover, reduced costs, improved efficiency, reduction in grievances, as well as 
increased quality and quantity of production, and improved morale". Each of these four areas 
are addressed and specific analysis questions are answered. A data analysis section provides 
information regarding the projects impact. Each of the four areas also includes a 
discussion/conclusions section which reviews the impact based on the data presented. 

LEVELS OF EXTERNAL EVALUATION VARIABLES 
FOR THE IMPACT STUDY 

OVERVIEW 

The purpose of the external evaluation according to the project proposal (page 24), is to (1) 
review the overall plan of the project and recommend any changes at the beginning of the 
project period, (2) review the design and implementation of the internal evaluation and to 
assess the appropriateness of the evaluation procedures for determining the effectiveness of 
the project, (3) conduct an external evaluation to determine the effectiveness of the project in 
meeting its overall objectives, and (4) prepare a final report of the impact of the project on 
"the participants and Georgetown Steel. In addition to the role description noted above, the 
external evaluator should provide suggestions and recommendations that will improve the 
overall project. The implementations of any recommendations and/or suggestions by the 
external evaluator are at the discretion of the Project Director and Staff. 

The purpose of this external evaluation plan is to provide a systematic strategy for the 
collection and analysis of data to determine the impact of the project. The format for this 
evaluation will include four levels of evaluation. The first level, the "Reaction" involves the 
instructor and participant reactions to aspects of the program. It answers the question, "How 
well did participants like the program 0 " 

The second level of evaluation, the "Learning" involves analysis of gains made by participants 
in knowledge and skills taught in the program courses. This is the academic achievement 
level of evaluation. For this project pre and post test score comparisons provide information 
on achievement gains. 
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The third level of evaluation, the "Behavior" involves changes in participant behavior. A 
survey of supervisory personnel provided specific information regarding the number 
of employees from each department who participated in the program; the number of 
participants who improved their communication skills; the number who increased their 
productivity; the number who improved attendance records; and the number who improved 
self esteem after participating in the program. The modified supervisor survey was used to 
collect this data. Participants were also be interviewed to determine their perceptions of 
behavior changes. 

The fourth level of evaluation, the "Results" involve such factors as reduced turnover, reduced 
costs, improved efficiency, reduction in grievances, as well as increased quality and quantity 
of production, and improved morale. Information provided by the Georgetown Steel Key 
Indicators was basis for the analysis of this level of evaluation. 

The next section deals with the internal evaluation plan, data collection procedures, and 
analysis of data. The section which follows the internal evaluation plan addresses specific 
details for each of the levels of the external evaluation impact study. 

THE INTERNAL EVALUATION 

The internal evaluation process, described on pages 33-36 of the proposal was implemented as 
described in the "Review of the Internal Evaluation Plan". A copy of that description follows 
below. 

REVIEW OF INTERNAL EVALUATION PLAN 
INPUTS: 

The internal evaluation plan (pages 33-36 of the Proposal) indicates the use of an Input, 
Process, and Output format for the evaluation of the project. The us*) of pre and post tests to 
measure literacy improvements in Reading and Mathematics are in place and the scores were 
used to validate any gains. The aggregation of item performance data could allow the project 
to compare overall scores on the tests as well as specific skill areas within the Math test such 
as whole numbers, fractions, percentages, or within the Reading test such as facts and details, 
finding information, following directions, and comprehension. The External Evaluation Plan 
provides the project with more detailed information on how to setup these data collection 
procedures. 

In order to increase confidence in the validity and reliability of the Math and Reading tests, it 
is recommended that the external evaluator be provided with student performance data in 
order to extend the work already done by Dr. Finkle. 
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Mastery levels on the pre-tests should be set at 85% and information on those employees who 
master the content on the pre-test sho d be retained along with information on those who 
score below that level. The recommendation of 85% as a cutoff on the pre-tests is based on 
an examination of the test items and the Individual Education Plans (IEP's). It was agreed 
that if an employee could score better than 85% that the employee did not need to participate, 
but they could do so if they so chose. 

According to the Proposal (pages 24 & 32) the project director should be given internal 
evaluation data from the Horry-Georgetown Director of Institutional Research and the 
Georgetown Steel Director of Industrial Relations. The partnership worked to set up a plan 
for the collection and analysis of internal evaluation data so that such information could be 
supplied to the project director providing the greatest utility possible. At this point in the 
project, the data collection needs to be standardized. The External Evaluation Plan will 
provide suggestions for such a data collection process. 

PROCESS: 

A listing of projected gains, benefits, and procedures was developed- based on. the information 
found in the proposal. This list was created by the Project Director and Dr. Finkle. This 
listing along with the Georgetown Steel Corporation Key Indicators and a review of the 
Project Objectives (see Appendix A of this document) provided the information needed to 
determine the process variable of the internal evaluation. 

OUTPUTS: 

The project proposal identifies benefits and anticipated gains as being (1) higher levels of 
literacy, (2) greater employee confidence and a higher level of self esteem, (3) improved 
productivity, (4) increased job security, (5) improved product quality, and (6) an improved 
workplace environment. 

Higher levels of literacy were determined using the results of the Reading and Mathematics 
pre and post test comparisons. Greater employee confidence and a higher level of self 
esteem and increased job security were determined by :z\ examination of the supervisor 
surveys and the number of project participants taking tests to seek higher pay jobs that require 
greater skills. Improved productivity and product quality were determined by an examination 
of the Georgetown Steel Key Indicators. This examination compared Key Indicators over the 
last three to four years to determine if during the project any of the Key Indicators improved. 
A survey of program graduates was designed to help determine if the project contributed to an 
improved workplace environment. 

The following is a list of outcomes identified in the internal evaluation section of the project 
proposal. These outcomes are related to the Output perspective of the internal evaluation. 
The outcomes are all indicators specified as possible effects of the program courses. 
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Rolling Mill Department (proposal p. 3 & 4) 

Outcome = Fewer finished goods that require re-work prior to release and delivery. 

Outcome = Less unscheduled down time and reduced equipment replacement costs. 
Steelmaking Department (proposal p. 3 & 4) 

Outcome = Faster change-over time from shift to shift 

Outcome = Improved delivery of steel to the next process step. 

Outcome = Growth in team process. 

Outcome = Increased "Job Ownership". 
Steelmaking - Mobile Equipment and Utilities Maintenance (proposal p. 4) 

Outcome = More efficient use of scheduled down time. 

Outcome = Safer work practices resulting in fewer accidents. 
Production, Planning, and Finishing Department (proposal p. 4) 

Outcome = Lower transportation end shipping costs. 

Outcome = Reduction of plant traffic. 

Outcome = More timely delivery of finished goods. 

Outcome = Costs associated with returned or damaged goods will be reduced. 

Outcome = Customer satisfaction will improve. 
Quality Assurance and D.R.I. Departments (proposal p. 4 & 5) 

Outcome = More accurate ongoing evaluations will be possible. 

Outcome = More timely information to affected departments. 
Overall (proposal p. 5 & 6) 

Outcome = Greater employee confidence. 

Outcome = Higher level of self-esteem. 

Outcome = Increased productivity. 

Outcome = Improved product quality. 

Outcome = Increased job security. . 

Outcome = Improved workplace environment. 

Information gathered by the internal evaluation must be provided on a timely basis to the 
external evaluator. The project staff will be responsible for supplying the external evaluator 
with the following data: 

1. Pre and Post Reading test scores in the format described in Appendix A 
for the purpose of additional analysis of reliability. 

2. Pre and Post Math test scores for the purpose of additional analysis of reliability. 

3. All supervisor survey data (see Appendix A ) The Revised New Horizons 
Quarterly Survey. 

4 All participant survey data (see Appendix A) New Horizons Participant Survey. 
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5. All project data required for the Full Data Base Development Format (see Appendix 
A) for specific information. 

6. Georgetown Steel Key Indicator data by quarter for the last three years - 1991, 1992, 
and 1993. 

7. Information on participants completion of job tests - including participant information 
by name, clock number, department, test taken, indication of pass or no pass, and 
outcome as to any job changes. 

8. Information on grievances filed by all employees over the last three to five years by 
quarter/those filed by New Horizons participants and the outcomes of each grievance. 

9. Information on the outcomes listed on page 5 of this document. 

EXTERNAL EVALUATION PLAN 

The external evaluation will focus on two factors as defined by the program proposal. These 
factors include (1) a review of the program objectives - to determine the extent to which each 
objective has been met by the program, (2) an "impact" study to determine the effectiveness 
of the project. 

The review of program objectives will include an analysis of each objective by the external 
evaluator and the development of a matrix. The matrix will include each objective, an 
indication of the degree to which that objective has been met, and the source of the data used 
to determine the effectiveness of the objective. Appendix A of this document contains an 
example matrix. 

The "impact" study to determine the effectiveness of the project will address the following 
four levels of evaluation. Each level listed below is defined, questions to be answered by 
data analysis are presented, and a description of the data sources for each question is 
presented. 

REACTION 

a. Reaction Defined - "Reaction" involves the instructor and participant 
reactions to aspects of the program. It answers the question, "How well 
did participants like the program?" 

b. Reaction Questions 

1. To what extent do participants and instructors feel that progress 
has been made toward academic goals? 

2. To what extent have participants achieved their own goals 0 

c Reaction Data Sources: Results of the New Horizons Participant Survey 
included in the Full Data Base. 
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Learning Defined - "Learning" involves analysis of gains nude by 
participants in knowledge and skills taught in the program courses 
Learning Questions 

1. To what extent have participants gronoi as readers? 

2. To what extent have participants grown as writers 0 

3. To what extent have participants grown in mathematics skills 0 

4. What are instructor perceptions of program effectiveness? 

5. To what extent do participants participate in establishing 
instructional goals and evaluating progress toward them 0 

Learning Data Sources: Pre and Post test scores for Reading, 
Mathematics and Writing included in the Full Data Base. Analysis of 
lEP's and feedback from New Horizons Participant Survey. Instructor 
Reactions to the Project Objectives Survey (found in the Site Visit 
Report dated June 26, 1993. 

Behavior Defined - "Behavior" involves changes in participant 
behavior. A survey of supervisory personnel will provide specific 
information regarding the number of employees from each department 
who have participated in the program; the number of participants who 
have improved their communication skills; the number who have ' 
increased their productivity; the number who have improved attendance 
records; and the number who have improved self esteem after 
participating in the program. 
Behavior Questions 

1 . To what extent has participants self-confidence as readers 
increased 0 

2. To what extent has participants self-confidence as writers 
increased? 

3. To what extent has participants self-confidence in mathematics 
skills increased? 

4. Has participants self-esteem increased 0 

Behavior Data Sources: The revised New Horizons Quarterly Survey 
and specific items on the New Horizons Participant Survey 

Results Defined - "Results" involve such factors as reduced turnover, 
reduced costs, improved efficiency, reduction in grievances, as well as 
increased quality and quantity of production, and improved morale. 
Results Questions 

1. To what extent has the participants earning power increased 0 

2. To what extent does the program show a return on the 
investment? 
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3. What is the rate of participation? 

4. To what extent does the program meet ne ds identified by 
external groups such as community and or job supervisors? 

5. What is the quality of recruitment and retention practices? 

6. To what extent does the program show evidence of coordination 
or collaboration with outside agencies and constituencies? 

Results Data Sources: Frequency of participant testing for new jobs 
within the company. Changes in the Georgetown Steel Key Indicators 
over a three year period by quarter. Project documentation and 
quarterly reports. Information tied to the 'Outcomes" listed on page 5 
of this document. 
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PROJECT IMPACT 
STUDY RESULTS 

SOME BASIC DEMOGRAPHICS 

During the grant funding period, 454 GSC employees completed the New Horizons training. 
Of the 747 GSC employees :he 454 trained represent 60.8% of the total number employed. 

Those involved in the training were from four different departments within GSC. The 
following identifies each of the departments, the number of total employees in each 
department, the number trained by the New Horizons project and the percentage in each 
department trained. 



Department 


Total 


Number 


Percent 


# Employees 


Trained 


Trained 


Hourly 








Rolling Mill 


177 


129 


72.8% 


Steel Making 


274 


191 


69.7% 


Shipping 


103 


47 


45.6% 


DRI 


60 


26 


43.3% 


NON-Hourly 








Admin, Sales 








Engineering, 








MIS. etc 


113 


6L 


45.8% 


TOTAL 


747 


454 


60.8% 



I 



Project participants had a wide range of education. The list below provides the highest grade 
completed by participants in grade ranges as well as the percent of those trained in each grade 
range group. 
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HIGHEST 

GRADE 

COMPLETED 



NUMBER OF PERCENT 
PARTICIPANTS 



0 to 6 Years 25 5% 

7 to 11 Years 79 1 8% 

12 Years 185 42% 

13 to 14 Years 88 20% 
15 to 16 Years 54 10% 
Above 16 Years _23 5% 
TOTALS 454 ' 100% 



The following list 


identifies the involvement 


of participants in New Horizons classes. 


CLASS 


U PARTICIPANTS 


% OF TOTAL 


Math 


56 


12% 


Reading 


61 


14% 


Writing 


13 


3% 


Time Mgt. 


122 


27% 


Prob. Solving 


202 


44% 


TOTALS 


454 


100% 



REACTION: 



To determine the reaction to the overall project, the two Reaction Questions were 
asked, and data was collected as indicated in the Reaction Data Sources below. 



a. Reaction Defined - "Reaction" involves the instructor and participant reactions 
to aspects of the program. It answers the question. "How well did participants 
like the program 9 " 

b. Reaction Questions 

1. To what extent do participants and instructors feel that progress has 
been made toward academic goals 0 

2. To what extent have participants achieved their own goals 0 

c. Reaction Data Sources: Results of the New Horizons Participant Survey 
included in the Full Data Base. 



Page 8 



REACTION DATA ANALYSIS: 



The New Horizons Participant Survey was given to a random sample of 108 project 
participants. A copy of the twelve item survey form can be found in the Appendix of this 
document. Each of the items on the survey could be responded to by marking SA for 
Strongly Agree, A for Agree, U for Undecided, D for Disagree, or SD for Strongly Disagree. 
Results of the survey for each item are presented below. 

ITEM PERCENT 

RESPONSE 

1. My communications skills have improved as a SA A U D SD 
result of my involvement tn the New Horizons 20% 71% 7% 1% 1% 
program. 

In response to item #1, 91% of the participants either strongly agreed or agreed that 
the program had helped to improve their communication skills. About 7% were 
undecided, and 2% disagreed with the item. 

2. My productivity has increased as a result of my SA A U D SD 
involvement in the New Horizons program. 16% 61% 17% 3% 3% 

In response to item #2, 77% of the participants either strongly agreed or agreed that 
the program had helped to improve their productivity. About 17% were 
undecided, and 6% disagreed with the item. 

3. My attendance record has improved as a result SA A U D SD 
of my involvement in the New Horizons program. 19% 28% 16% 28% 9% 

In response to item #3, 47% of the participants either strongly agreed or agreed that 
the program had helped to improve their attendance. About 16% were undecided, and 
37% disagreed with the item. (Based on Supervisor Survey data, attendance has never 
been a problem at GSC.) 

4 My self esteem has improved after taking part SA A U D SD 
in the New Horizons Program 25% 68% 5% 0% 2% 

In response to item #4, 93% of the participants either strongly agreed or agreed that 
the program had helped to improve their self esteem. About 5% were undecided, and 
2% disagreed with the item. 
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5. I enjoyed the training classes. 



SA A U D SD 

39% 60% 0% 0% 1% 



In response to item #5, 99% of the participants either strongly agreed or agreed that 
they had enjoyed their training classes. None were undecided, and 1% disagreed with 
the item. 

6. I learned a lot from my instructor. SA A U D SD 

31% 66% 2% 0% 1% 

In response to item #6, 97% of the participants either strongly agreed or agreed that 
they had learned a lot from their instructor. About 2% were undecided, and 1% 
disagreed with the item. 

7. I reached the goals I set in my IEP. SA A U D SD 

. 13% 67% 11% 4% 5% 

In response to item #7, 80% of the participants either strongly agreed or agreed that 
they had reached the goals they set in their IEP's. About 1 1% were undecided, and 
9% disagreed with the item. 

8. My New Horizons training has given me the SA A U D SD 
self-confidence to take tests for jobs that pay 20% 60% 8% 7% ?% 
more. 

In response to item #8, 80% of the participants either strongly agreed or agreed 
that the program had helped to improve their self-confidence. About 8% were 
undecided, and 12% disagreed with the item. 

9. I liked the awards given when I completed SA A U D SD 
my training. 23% 68% 5% 2% 2% 

In response to item #9, 91% of the participants either strongly agreed or agreed that 
they liked the awards given when they completed their training. About 5% were 
undecided, and 4% disagreed with the item. 

10. My New Horizons training has helped make SA A U D SD 
me a better worker. ^ 12% 76% 10% 1% 1% 

In response to item #10, 88% of the participants either strongly agreed or agreed that 
the program had helped make them better workers. About 10% were undecided, and 
2% disagreed with the item. 
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11. Having my training during working hours SA A U D SD 
was great for me. Better than if the training 34% 46% 8% 5% 7% 
had been after work on my own time. 

In response to item #11, 80% of the participants either strongly agreed or agreed that 
having the training during working hours was better than having the training after 
working hours. About 8% were undecided, and 12% disagreed with the item. 

12. I have a better work environment because SA A U D SD 
of my New Horizons training. 13% 69% 14% 3% 1% 

In response to item #12, 82% of the particip vt s either strongly agreed or agreed that 
the program had helped create a better work environment. About 14% were 
undecided, and 4% disagreed with the item. 

* Charts of the Participant Survey results for each item can be found in the Appendix B of 
this document, (pages 41 to 52) 

DISCUSSION/CONCLUSIONS - REACTION: 

For all four departments involved in the training, high percentages of participation were 
reached. Overall more than 52% of GSC employees were involved in the training. Some 
35% of the participants were involved in the Reading, Writing and Mathematics classes. The 
majority of those who had completed 12 years of schooling or less were involved in the 
Reading, Writing and Mathematics classes. About 70% of those involved in the training had 
12 years or less of schooling. 

In response to the Participant Survey, 91% reported that their communications skills had 
improved. They also reported increases (77%) in their productivity as result of beinr 
involved in the New Horizons project. Since attendance is veiy high at GSC item # 3 did not 
provide significant results although 47% agreed or strongly agreed with the item. 
Regarding increases in self-esteem, 93% reported increases. Many of these reactions seem to 
be validated by the Quarterly Survey results found in the Behavior section of this document. 

Participant reactions to the training classes was very positive. Some 99% agreed or strongly 
agreed that they enjoyed the training classes. About 97% reported learning a lot from their 
instructors, and 80% felt that they had reached the goals set in their lEP's. 
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Many reported increases in self-confidence to take tests for better jobs (80%). This seems to 
be reflected in the actual increases in the number of participants taking tests for better jobs. 
Those numbers increased from 31 in 1991, to 54 in 1992, to 73 in 1993. In addition, the 
percentage of those who passed the tests increased from 55% in 1991, to 78% in 1993. 
More specific information regarding participants who took tests for better job.; can be found 
in the Results section of this document. 
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Participants liked the awards program that was part of the project (91%). They felt (88%) 
that they were better workers because of the New Horizons training. Many (80%) agreed or 
strongly agreed that "having the training during working hours was a posuive aspect of the 
program. They felt that this was better than if the training had been after work or on their 
own time. They reported that their work environment was better because of the training 
(82%). 

All indications are that at participants liked the program and thought it was very useful. This 
is confirmed by interviews with instructors. In answering the question "How well did 
participants like the program?", the answer is very well. According to the data collected in 
the Participant Survey, answers to the Reaction questions, (1.) To wiiat extent do participants 
and instructors feel that progress has been made toward academic goals 0 and (2.) To what 
extent have parti chants achieved their own goals 0 seems to be to a great extent. Evidence 
supporting these conclusions is provided by responses to item #7 "I reached the goals I set ir. 
my IEP". In response to item #7, 80% of the participants either strongly agreed or agreed 
that the program had helped to improve their communication skills. About 11% were 
undecided, and 9% disagreed with the item. 

Regarding the Reaction aspect of this evaluation, the conclusion is that participants liked the 
program very much and from that aspect the project was a resounding success. 



i 
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LEARNING 

a. Learning Defined - "Learning" involves analysis of gains made by 
participants in knowledge and skills taught in the program courses 

b. Learning Questions 

1. To what extent have participants grown as readers 0 

2. To what extent have participants grown as writers 0 

3 To what extent have participants grown in mathematics skills? 

4. What are instructor perceptions of program effectiveness 0 

5. To what extent do participants participate in establishing 
instructional goals and evaluating progress toward them? 

c. Learning Data Sources: Pre and Post test scores for Reading, 
Mathematics and Writing included in the Full Data Base. Analysis of 
IEP's and feedback from New Horizons Participant Survey. Instructor 
Reactions to the Project Objectives Survey (found in the Site Visit 
Report dated June 26, i993). 

LEARNING DATA ANALYSIS: 

Many of the New Horizons Participants were involved in Reading, Mathematics, and Writing 
classes. In order to answer the learning questions stated above, participants were given pre 
and* post tests in reading, mathematics, and writing. The development and validation of these 
tests is described in the other project documents. The following is an overview of the 
reliability study data for the both the Reading and the Mathematics tests. The validity study 
is discussed in the Site Visit Report for September 9th & 10th, 1993. 

ALPHA TEST OF INTERNAL CONSISTENCY FOR GSC READING PRE-TEST 
RELIABILITY ANALYSIS - SCALE (ALPHA) 
Reliability Coefficients 

N of Cases = 50.0 N of Items = 47 

Alpha - .8821 

ALPHA TEST OF INTERNAL CONSISTENCY FOR THE GSC MATH PRE-TEST 
RELIABILITY ANALYSIS - SCALE (ALPHA) 
Reliability Coefficients 

N of Cases - 43.0 N of Items = 46 

Alpha - .8879 

What follows is an analysis of pre and post test comparisons for each of the three areas of 
training. 
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MATHEMATICS PRE AND POST TESTS: 

Analysis Question = To what extent have participants grown in mathematics skills? 
This question was answered by conducting a pre to post test comparison with a paired groups 
t-test. The t-test was used to determine if there were significant differences between the mean 
pre and post test scores. This analysis was conducted using 57 of the participants. 



Pre-Test Mean Score 58.98 

Post-Test Mean Score 82.21 

Mean Difference(Gain) 23.23 

t-value 12.28 

Degrees of Freedom 56 

2-tail Significance p.> 0001 



The total score gain based on the pre test score was 23.23 points or a 39.4 % in<. ^ase from 
the pre to the post test. 

READING PRE AND POST TESTS: 

Analysis Question = To what extent have participants grown in reading skills 0 
This question, was answered by conducting a pre to post test compaiison with' a paired groups 
t-test. The t-test was used to determine if there were significant differences between the mean 
pre and post test scores. This analysis was conducted using 61 of the participants. 



Pre-Test Mean Score 81.67 

Post-Test Mean Score 89.36 

Mean Difference(Gain) 7.69 

t-value 5.67 

Degrees of Freedom 60 

2-tail Significance p>.0001 



Given that several of those involved in the reading instruction had scored 85% or higher on 
the pre test, a second analysis was conducted where participants with pre test scores at 85% 
and above were removed. This analysis was conducted using 28 of the participants. The 



results of this analysis are listed below. 

Pre-Test Mean Score 69.17 

Post-Test Mean Score 84.57 

Mean Difference(Gain) 15.40 

t-value 7.46 

Degrees of Freedom 27 

2-tail Significance p.>.0001 



The total score gain based on the pre test score was 15.40 points or a 22.3 % increase from 
the pre to the post test. 
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WRITING PRE AND POST TESTS: 

Analysis Question = To what extent have participants grown in writing skills 9 
This question was answered by conducting a pre to post test comparison with a paired groups 
t-test. The t-test was used to determine if there were significant differences between the mean 
pre and post test scores. This analysis was conducted using 19 of the participants. 



Pre-Test Mean Score 58.05 

Post-Test Mean Score 66.52 

Mean Difference(Gain) 8.47 

t-value 4.39 

Degrees of Freedom 18 

2-tail Significance p.>.0001 



The total score gain based on the pre test score was 8.47 points or a 14.6 % increase from 
the pre to the post test. 

* Charts of the Math, Reading and Writing results be found in the Appendix 3 of this 
document, (pages 53 to 56} 

INSTRUCTOR AND STAFF PERCEPTIONS 
OF PROGRAM EFFECTIVENESS: 

Analysis Question = What were staff perceptions of program effectiveness? 
To determine, "What were staff perceptions of program effectiveness 0 " six of the project staff 
members including administrators and instructors were asked to complete the Project 
Objectives Survey. The purpose of the survey was to determine the degree to which project 
objectives had been met. The results of this survey are presented below. 

MET? KEY 

MET = OBJECTIVE MET IN FULL OR FOR SPECIFIC PROJECT 

COMPONENTS AT THE TIME OF THIS REPORT. 
IP = IN PROGRESS - ON GOING AND WHERE EXPECTED AT 

THIS POINT IN TIME 
NM = NOT MET 

SOURCE OF DATA KEY 

PR = Participant Records 

QR = Project Quarterly Reports 

PI = Participant Interviews 

SI - Staff Interviews 

SR = Participant Survey Results 

POS = Project Objective Survey 

IMR = Project Materials Review (Instructional Materials) 

SVR = Site Visit Reports 
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Objective 


Met° 


Source of Data 


1.1 


To identify and select instructors who have a 
successful background in Adult Education and 
job-related workplace training. 


MET 


PR, SI, QR, POS 


1.2 


To conduct training and orientation sessions for 
all instructors, which will: review effective 
instructional techniques, adult learning theories, 
program goals and objectives, course goals and 
objectives, Individualized Educational Plans and 
program policies and procedures. 


MET 


PR, SI, POS 


1.3 


To conduct, throughout the project, ongoing 
assessments and evaluations of all instructors. 


MET 


PR, SI, QR, POS,SVR 


2.1 


To conduct and complete comprehensive needs 
assessments to verify the workers who require 
basic skills training ?nd to verify the specific 

1 i , l 1_ 1*1" _ _ 1 _ _ _ i 

job related basic skihs required for each 
participant to most effectively perform his/her 
job and become more productive. 


MET 


IMR, PR, PI, SI, POS 


2.2 


To develop competencies (based on the needs 
assessments) for each identified skill. 


MET 


IMR, SI, POS 


3.1 


To write measurable performance objectives. 


MET 


IMR, SI, POS, SVR 


3.2 


To develop course measurable objectives . 


MET 


IMR, SI, POS, SR 


4.1 


To develop course objectives and course 
outlines. 


MET 


IMR, SI, POS, SVR 


4.2 


To design course content to reflect the 
competencies and skills taught. 


MET 


IMR, QR, SI, POS 


4.3 


To develop job related curriculum materials, 
based on the Literacy Task Analysis, that 
enhance adult learning and which are designed 
to meet the stated competencies and 
performance objectives. 


MET 


IMR, SVR, SI, POS 


5 ] 


To develop a written Individualized Educational 
Plan for each participant. The Individualized 
Educational Plans will be negotiated and 
developed by the participants and the 
instructors. 


MET 


PR, IMR, SR, POS 
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6.1 


To conduct basic skills training for at least 484 
("rpnrpptnvvn Stppl pmnlovees dunne the oroiect 
period. 


MET 


PR, QR, POS j 


6.2 


To offer classes and instruction in: Math and 

RpnHinu Writinw Prohlpm SolvinP and 

Decision Making , and Time Management . 


MET 


IMR, QR, SI, SR, 
POS 


6.3 


To offer a program option of one-on-one 

+iitorincr for triocf* \x/itri nri or PvtrPtnPlv low 
lUlUIlUg 1UI UlUoC Willi ii\)y KJi caii cuiciy ixjw, 

reading skills. 


MET 


IMR, PR, SI, POS 


6.4 


To provide high school completion/GED 
preparation classes and training to all who 
desire, or need, to reach this educational level. 


ME1 


PR, SI, POS 


6.5 


To provide class schedules and class sizes 
which will meet the needs of all participants 
and enhance learning. Class sizes will be 12-15 
students per class. Classes will be offered 
Monday through Friday. Four class sessions 
will be offered each day: 7:00 a.m. - 9:00 a.m., 
9:00 a.m. - 11:00, 1:00 p.m. - 3:00 p.m. and 
3:00 p.m. - 5:00 p.m. Two classes of 12-15 
students per class will be held during each two- 
hour session. Because workers work on a 
regular schedule of rotating shifts, these 
sessions will accommodate the needs of all 
participants. 


MET 


PR, SI, POS, IMR, 
SR 


6.6 


To provide an opportunity for every participant 
to achieve his/her basic skills 
improvement goals. Each participant will be 
scheduled two hours per day (ten hours per 
week) until the competency level objectives 
defined in the Individualized Educational Plan 
have been attained. The hours of training 
completed by each worker may range from 30 
hours to 100+ hours depending upon the basic 
skills competencies needed. As participants 
attain their competencies, others will be 
scheduled for training. 


MET 


IMR, SI, POS, SR, 
PR 
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7.1 


To conduct a comprehensive evaluation of 
trainee outcomes as well as an evaluation of the 

training nrnoram 


MET 


QR, SVR, PR, POS 


8.1 


To develop and implement a multifaceted media 

campaign to promote the project and 

inform the community about the benefits of the 

program. 


MET 


QR, SI, POS 


8.2 


The Project Director will generate a quarterly 
report which will be submitted to key project 
personnel. Appropriate information included in 
the report will be made available for press 
releases and public service announcements. 


MET 


QR, PR, POS 


8.3 


Upon completion of the first year, at least two 
articles will be submitted to journals in 
the fields of Adult Education and Industrial 
Management. Articles will describe the project 
and include information beneficial to other 
corporations and colleges considering similar 
relationships. 


IP 


SI, POS 


8.4 


Upon completion of the project, the Director 
will: (1) submit a Final Report to the USD. of 
E., (2) hold a news conference summarizing the 
project and its benefits, (3) submit at least three 
articles to three journals with national 
circulations, (4) hold a conference open to the 
public, other agencies, corporations and 
institutions, and (5) disseminate a summary of 
the project and copies of published articles to 
the South Carolina Work Force Initiative 
Program Specialists. 


IP 


SI, POS 
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PARTICIPANT INVOLVEMENT IN 
ESTABLISHING GOALS AND EVALUATING 
PROGRESS TOWARD THOSE GOALS: 

Analysis Question = To what extent do participants participate in establishing 

instructional goals and evaluating progress toward them? 

To determine, "To what extent do participants participate in establishing 

instructional goals and evaluating progress toward them 0 ", Program participants were 

surveyed after they completed the training. Specific items from the Participant Survey Part II 

address this question. The results of this survey are presented below. 

For each of the items on the second part of the Participant Survey that relate to this Analysis 
Question, the percentage of response is given to each item option. The analysis results are 
listed below by item. This survey was completed by the participants who were involved in 
the training. 



What did you hope to achieve by taking classes from the New Horizons 
Program 9 % After 

Training 

a. To upgrade my skills 27% 

b. Self improvement 22% 



Other Please explain below. 



51% 



(For Example, Learn more, get a better job, etc.) 
2. Did the program meet your goals from number 1 above? 

% After 
Training 

a. Yes 78% 

b. No 5% 

c. Yes and No Please explain below.... 17% 

(For Example, Both, for the most part, almost all. etc.) 
D ISCUSS ION7CONCLUS IONS - LEARNING: 

A great deal of effort during the early phases of the project went into the development and 
validation of the Mathematics, Reading and Writing classes and assessment instruments to 
measure student growth from pre to posttest. This effort did pay off for the project. 
Instrument validation studies indicate that the assessment instruments used were both valid 
and reliable. Specific analysis of the reliability of the reading and mathematics instruments 
yielded internal consistency for both instruments at .88 or higher (see Alpha Test information 
Page 13 above). Validity was determined using the development process and one-to-one 
correspondence between objectives and the study conducted by the internal evaluator. 
Reliability of the writing assessment was determined by comparisons of the raters and the 
validity was determined based on a one-to-one match to provide some evidence of face and 
content validity. 
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Learning involves analysis of gains made by participants in knowledge and skills taught in the 
program courses. The questions for the Learning aspect of the evaluation were (1) To what 
extent have participants grown as readers?, (2) To what extent have participants grown as 
writers?, (3) To what extent have participants grown in mathematics skills?, (4) What are 
instructor perceptions of program effectiveness?, (5) To what extent do participants participate 
in establishing instructional goals and evaluating progress toward them?. The Learning Data 
Sources were: Pre and Posttest scores for Reading, Mathematics and Writing, as well as 
analysis of IEP's, feedback from New Horizons Participant Survey, and instructor/staff 
reactions to the Project Objectives Survey. 

Gains made by participants in Math as measured by pre and posttest comparisons were 
statistically significant. The mean gain was 23.23 points yielding a t-value of 12.28, where 
p.< .0001. In Reading gains by participants were measured by pre and posttest comparisons 
and were found to be statistically significant. The mean gain was 15.40 points yielding a t- 
value of 7.46, where p.< 0001. For the Writing gains, pre and posttest comparisons were 
found to be statistically significant. The mean gain was 4.39 points yielding a t-value of 
4.47, where p.< .0001. The answer to the first three Learning Questions is that statistically 
significant gains were made by participants in Math, Reading and Writing. 

Instructor/staff perceptions of program effectiveness were measured by responses to the 
Project Objective Survey. Six staff members responded and the results (found above) indicate 
that the project has met 20 of the 22 program objectives. The two objectives not yet met are 
in progress toward being met and can only be satisfied after the project is completed. For 
each project objective, sources of data are identified and the objectives are marked MET, 
NOT-MET or IP for In Progress. According the responses to the Project Objective Surrey, 
and an analysis of identified data sources, the answer to question #4 "What are instructor 
perceptions of program effectiveness?" is that the instructors and staff felt that the program 
was very effective. Regarding Learning Question #5, "To what extent do participants 
participate in establishing instructional goals and evaluating progress toward them?", the 
answer based on an analysis of paiticipant records is that participant were highly involved. In 
addition participant reactions to the Participant Survey, item #7 indicate that they agreed, 
(80%) or strongly agreed that they reached the goals they set in their IEP's. 

Overall, the analysis of the learning aspect of the evaluation provides evidence that significant 
increases in Reading, Math, and Writing were achieved by participants. These gains were 
documented using pre and posttest comparisons and a t-test to determine significant 
differences of the gains. In addition, the Survey responses by staff and participants indicate 
project objectives were met and participants did achieve goals set in their IEP's. 
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BEHAVIOR 

a. Behavior Defined - "Behavior" involves changes in participant 
behavior. A survey of supervisory personnel will provide specific 
information regarding the number of employees from each department 
who have participated in the program; the number of participants who 
have improved their communication skills: the number who have 
increased their productivity; the number who have improved attendance 
records; and the number who have improved self esteem after 
participating in the program, 

b. Behavior Questions 

1. To what extent has participants self-confidence as readers 
increased? 

2. To what extent has participants self-confidence as writers 
increased? 

3. To what extent has participants self-confidence in mathematics 
skills increased? 

4. Has participants self-esteem increased? 

c. Behavior Data Sources: The revised New Horizons Quarterly Survey 
and specific items on the New Horizons Participant Survey 

BEHAVIOR DATA ANALYSIS: 

SUPERVISORS 

To determine the impact of participant behavior changes related to the training, two sources 
of information ware used. First, supervisors were interviewed each quarter of the project and 
asked questions found on the New Horizons Quarterly Survey (see Appendix of this document 
for a copy of the survey). Second, specific items were included on the Participant Survey to 
determine answers to the following Analysis Questions: 

1. How many participants have improved their communications skills? 

2. How mar ^ participants have increased their productivity 0 

3. How many participants have improved attendance records 0 

4. How many participants do you feel have improved self esteem 
after participating in the New Horizons Program 0 

Taken one at a time, the supervisor responses to these questions provide evidence of the 
numbers and percentages of participants whose behaviors changed on the job. 
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For question #1, "How many participants have improved their communications skills? " 

Supervisor interviews indicated that overall, 56% of their employees who had completed the 
training had improved communication skills. The supervisors mentioned the following 
indicators most frequently, as examples of increased communication skills by program 
participant. 

"Can report information from the field better..." 

"More open than they were before - working better with the supervisor" 
"They ask more questions..." 

"More interaction with other employees from other departments and 

seem to communicate more with each other about work in the department..." 

For question #2, "How many participants have increased their productivity? " 

Supervisor interviews indicated that overall, 47% of their employees who had completed the 
training had improved productivity. The supervisors mentioned most frequently the following 
indicators as examples of increased productivity they could see changes in on the job. 

"Take more responsibility and start more projects on their own..." 

"Being in the classes has made them be more aware of their jobs..." 

"Improved ability to do work more efficiently..." 

"Faster change over from shift to shift..." 

"The team is working more smoothly than before..." 

For question #3, "How many participants have improved attendance records?" 

Supervisor interviews indicated that overall, their employees attendance records were already 
very good. They did not see any marked change in attendance. 

This information from the supervisors would seem to be reflected in the responses of the 
participants to the survey. Note the percentages of response to the following survey question: 

For question #4, "How many participants do you feel have improved self esteem after 
participating in the New Horizons Program ?" 

Supervisor interviews indicated that overall, 70% of their employees who had completed the 
training had improved self esteem. The supervisors mentioned most frequently the following 
indicators as examples of increased self esteem they could see changes in on the job. 

"They are more responsible and take more pride in their work" 
"They start more work related discussions with co-workers" 
"They are more open to discuss department operations in general" 
"Contribute to the team more than before..." 
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PARTICIPANTS 

This information from the supervisors would seem to be reflected in the responses of the 
participants to the survey. Note the percentages of response to the following survey question: 

To answer the following Analysis Questions, data was collected from several sources. First, 
the improved test results for reading, math, and writing would provide some indication that 
participants have become better learners. Responses to Participant Survey also provide some 
indication of their self-confidence and self-esteem. 

1. To what extent has participants self-confidence as readers increased 0 

2. To what extent has participants self-confidence as writers increased? 

3. To what extent has participants self-confidence in mathematics skills increased 0 

4. Has participants self-esteem increased? 

Participant Survey Item Percent of Response 

1. My communications skills have improved as a SA A U D SD 
result of my involvement in the New Horizons 20% 71% 7% 1% 1% 
program. 

In response to item #1, 91% of the participants either strongly agreed or agreed that the 
program had helped to improve their communication skills. About 7% were undecided, and 
2% disagreed with the item. 

2. My productivity has increased as a result of my SA A U D SD 
involvement in the New Horizons program. 16% 61% 17% 3% 3% 

In response to item #2, 77% of the participants either strongly agreed or agreed that the 
program had helped to improve their produtivity. About 17% were undecided, and 6% 
disagreed with the item. 

4. My self esteem has improved after taking part SA A U D SD 
in the New Horizons Program 25% 68% 5% 0% 2% 

In response to item #4, 93% of the participants either strongly agreed or agreed that the 
program had helped to improve their self esteem. About 5% were undecided, and 2% 
disagreed with the item. 
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DISCUSSION/CONCLUSIONS - BEHAVIOR: 

To determine behavior changes of participants, interviews with supervisors were conducted. 
The basis for the interviews was the Quarterly Survey Form. Each quarter, the staff used the 
form to help them interview supervisors from each of the four GSC departments. Each 
supervisor interviewed. had employees who were involved in the training, (see Appendix A 
for a copy of the form used.) 

The idea behind the interviews was to determine changes in participant behavior on the job. 
The information previously presented indicates focus on four questions asked during the 
interviews. Overall, the supervisors reported about 56% of the participants had improved 
their communication skills and the supervisors were able to point out specific indicators 
demonstrated by participants. They also reported about 47% of the participants had increased 
productivity. Related to improved self-esteem, the supervisors reported that about 70% 
having improved self-esteem. Attendance was not seen as a problem by the participants 
or the supervisors. 

The comparison of participant responses to those of the supervisors shows that in fact both 
groups saw improvements in three of the four areas. For increases in communication skills, 
the supervisors reported 56%, while the participants reported 91%. In the area o r 
productivity, the supervisors reported 47% while the participants reported 77%. ror 
improvements in self-esteem, the supervisors reported 70% while the participants reported 
93%. Although one would not expect these figures to agree 100%, the high positive direction 
of the estimates does suggest that changes of a positive nature did, in fact, take place. One of 
the important conclusions involves the fact that supervisors were able to describe specific 
behaviors that they saw as indicators of change. These indicators could be refined and used 
in the future to help determine behavior changes that are related to the training. 

There is positive evidence to support the proposition the positive behaviors see in the work 
place were influenced by the New Horizons training. 
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RESULTS: 



a. Results Der.ned - "Results" involve such factors as reduced turnover, reduced costs, 
improved efficiency, reduction in grievances, as well as increased quality and quantity 
of production, and improved morale. 

b. Results Questions 



1. 


Is there greater employee confidence? 


2. 


Do participants exhibit higher levels of self-esteem 9 


3. 


Is there increased productiv : ty ? 


4. 


Has product quality improved ? 


5. 


Do employees have increased job security 9 


6. 


Has the workplace environment improved ? 


7. 


Has safety improved ? 


8. 


Has there been a reduction in grievences ? 



c. Results Data Sources: Frequency of participant testing for new jobs within the 

company. Changes in the Georgetown Steel Key Indicators over a three year period 
by quarter. Project documentation and quarterly reports. Information tied to the 
'Outcomes" listed on page 5 of this document. 

RESULTS DATA ANALYSIS: 

To determine the impact of project results related to the training, several sources of 
information ware used. First, changes in Georgetown Steel Key Indicators over a three year 
period were examined. These Key Indicators are related to the outcomes listed in the project 
proposal and reproduced below. Second, the frequency of participant testing for new jobs 
over the past three to four years. 

RESULTS ANALYSIS QUESTIONS: 

OVERALL: 

According to the project proposal (proposal p. 5 & 6), overall desired results of the project 
included the following outcomes: 



1. 


Greater employee confidence. 


2. 


Higher level of self-esteem. 


3 


Increased productivity. 


4. 


Improved product quality. 


5 


Increased job security 


6. 


Improved workplace environment. 


7 


Improved Safety 


8. 


Reduction in Grievences 
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OUTCOME U\ - Greater Employee Confidence 

This outcome was measured by item #10 from the Participant Survey 

My New Horizons training has helped make SA A U D SD 
me a better worker. ~ 12% 76% 10% 1% 1% 

88% of the participants either strongly agreed or agreed that the program had helped to 
make them better workers. About 10% were undecided, and 2% disagreed with the 
item. 

OUTCOME #2 - Higher levels of self-esteem. 

This outcome was measured by responses to the Participant Survey and Supervisor 
responses to the Quarterly Survey. 

Participant Survey 

My self esteem has improved after taking part SA A U D SD 
in the New Horizons Program. 25% 68% 5% 0% 2% 

93% of the participants either strongly agreed or agreed that the program had helped to 
improve their self esteem. About 5% were undecided, and 2% disagreed with the 
item. 

Quarterly Survey - Suprrvisois 

"How many participants do you feel have improved self esteem after participating in 
the New Horizons Program ° M 

Supervisor interviews indicated that overall, 70% of their employees who had 
completed the training hac- improved self esteem. 

OUTCOME #3 - Increased Productivity 

This outcome was measured by responses to the Participant Survey, Quarterly Survey, 
and the GSC Key Indicators. 

Participant Survey 

My productivity has increased as a result of my SA A U D SD 
involvement in the New Horizons program. 16% 61% 17% 3% 3% 

77% of the participants either strongly agreed or agreed that the program had helped to 
improve their productivity. About 17% were undecided, and 6% disagreed with the 
item. 

Quarterly Survey - Supervisors 

"How many participants have increased their productivity 0 " 
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Supervisor interviews indicated that overall, 47% of their employees who had 
completed the training had improved productivity. 



OUTCOME U4 - Improved product quality. 

This outcome ,vas measured by GSC Key Indicator reporting Customer Claims for 
Quality in dollars over a three year period. In addition, Billets scrapped and Coils 
scrapped for quality reasons were also examined over a three year period. 



GSC Key Indicators 

The GSC Key Indicators listed below were identified in the project proposal as 
outcomes that were defined as indicators of productivity. 

YEAR 



1993 

58.4 
93.6 
3.22 
4.75 



iNDICATOR 1991 1992 

Malt Shop Production Rate- (tph) 51.7 55.5 

Rolling Mill Production Rate (tph) 91.0 87.9 

Melt Shop % Maintenance Delay 3.63 3.37 

Rolling Mill % Maintenance Delay 5.52 3.70 
GSC Key Indicators 

YEAR 

INDICATOR 1991 1992 

Customer Claims for 

Quality in dollars $856,269 $581,604 $242,283 



1993 



% Increase or 
Reduction 

+13% 
+3% 
-11% 
-14% 



%Reduction 



72% 



Coils Scrapped 

for Quality Reasons(%) 



1.2 



2.0 



1.0 



[7% 



OUTCOME US - Increased job security. 

This outcome was measured by frequency of participant testing for better jobs, within 
GSC. The information below describes the number of tests taken to qualify for 
better jobs within GSC by participants of the New Horizons Project from 1991 
through 1993. In addition to the number of tests taken, the number passed and failed 
as well as the percentage passed and failed is presented. 



YEAR 


# TESTS 


#F AILED 


^PASSED 


%PASSED 


%F AILED 




TAKEN 










1991 


31 


14 


17 


55% 


45% 


1992 


54 


6 


48 


89% 


1 1% 


1993 


73 


16 


57 


78% 


22% 



Percent increase 1991 to 1993 = 135% 
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OUTCOME #6 - improved woritplace environment 

This outcome was measured by responses to the Participant Survey item #12. 



Participant Survey 

I have a better work environment because 
of my New Horizons training. 



SA 
13% 



U 



D 



SD 



69% 14% 3% 1% 



82% of the participants either strongly agreed or agreed that the program had helped to 
improve their work environment. About 14% were undecided, and 4% disagreed 
with the item. 



i 
i 



OUTCOME til Improved Safety 

Other specific outcomes related to the Results aspect of the data analysis include: 
OUTCOME #7 - Safer work practices resulting in fewer accidents. This outcome 
can be verified through an analysis of the following GSC Key Indicators. 

YEAR 

INDICATOR 1991 1992 1993 



Lost Time Accidents 33 
Compensibles 134 

OUTCOME #8 Reduced Grievences 



106 



6 

120 



%Reduction 
Over 1991 
82% 



7o 



Another outcome related to the Results aspect of the evaluation is the number of 
grievances filed by employees over a three to four year period. The assumption here 
is that if employees gain improved communication skills, that an increased number of 
problems will be resolved before they reach the grievance level. This would also be 
related to an improved workplace environment. The information below represents the 
number of grievances filed during the last nine years. 

Number of Grievances Filed 
1985 1986 1987 1988 1989 1990 1991 1992 1993 
152 106 108 97 105 129 60 25 55 

The pattern of grievances presents an interesting question. Has involvement in the 
New Horizons project and increased communication skills helped to reduce 
grievances? During the project years, the number of grievances was reduced by 57% 
in 1993 when compared to 1990, and by 81% in 1992 when compared to 1990. 
There was also a reduction in 1991, but that was not a project year. The overall trend 
from 1985 to 1990 averaged 1 16 grievances, while the trend from 1991 to 1993 
averaged 47 grievances. Another interesting fact is that the projected number of 
grievances for 1994 is 22. That is based on the 1 1 grievances filed through June 



i 
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of 1994. The assumption here is that responses from the supervisors and the 
participants would indicate that the increased levels of communication have influenced 
the reduction of documented grievances filed. 

DISCUSSION/CONCLUSIONS - RESULTS: 

Analysis of the data related to the Results aspect of the project evaluation indicates that 
participants as well as supervisors reported increases in employee confidence, higher levels of 
self-esteem, and increased productivity. Regarding productivity, specific GSC Key indicators 
reflect a reduction of 1 1% in maintenance delays for the Melt Shop and a reduction of 14% in 
maintenance delays for the Rolling Mill when comparisons are made between figures from 
1991 and those for 1993. The Melt Shop increased their production 13% from 1991-1993 
while the Rolling Mill increased their production 3% over the same period. 

With respect to improved product quality, customer claims for quality (in dollars) saw a 72% 
reduction from 1991 to 1993. The percentage of coils scrapped for quality reasons was 
reduced by 17%. 

One indication of "job security" used by the project was the number of tests taken by 
participants to qualify them for better jobs (higher paying). From 1991 to 1993 
there was a 135% increase in the number of tests taken by participants to qualify for better 
jobs. In addition to this dramatic increase, in the number of tests taken, the passing rate 
increased from 55% in 1991 to 78% in 1993. This information taken with the fact that the 
number of tests taken by Non-Participants remained about the same over the three year 
period according the records provided by the GSC office of personnel, provides support 
for the idea that the training influence increased participant willingness to take the tests and 
that the passing rate increased. 

GSC Key Indicators related to safety include the number of lost time accidents and the 
number of compensible accidents. An analysis of these figures yields an 82% reduction in 
lost time accidents over the period 1991 to 1993, and an 1 1% reduction in the number of 
compensible accidents over the same time period. 

Impact related to the Results aspect of the evaluation seems to show clear trends of change 
for the better. Since it is virtually impossible to draw cause and effect relationships between 
the training and overall steel production, it is important to examine the trend of key 
indicators. One safeguard in this area of evaluation to protect for over generalization is to 
document that for the areas of GSC where project participants work that no new equipment 
has been installed that would impact some of the variables examined. In addition it is 
important to document that no drastic changes in plant process have been implemented by 
management during this time period. Information gained from MIS suggests that none of 
these changes have been instituted during the period of evaluation. Changes that are 
instituted by teams on the mill floor however, are acceptable. Given the fact that the project 
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trained some 53% of the GSC work force, these Results are even more powerful. 

There is no doubt that there has been greater employee confidence, a higher level of self- 
esteem among participants, increased productivity, improved product quality, a feeling of 
increased job security among participants, perception of improved workplace environment, 
improved safety, and a reduction in grievances filed during the project period as measured by 
the outcomes and indicators identified in the project proposal. All of these facts indicate a 
very high level of project success. In addition, the impact of the project on GSC has been 
very positive, with tneasureable improvements in attitude as well as productivity. 
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APPENDIX A: "FORMS" 



Participant Survey Form 
Analysis Matrix of Project Objectives 
Full Data Base Development Format 
New Horizons Quarterly Survey - (Superviso 



i 

E 



i 
1 
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Name 



NEW HORIZONS PARTICIPANT SURVEY 

Clock Number 



Directions: Read each statement below and then circle "SA" if you Strongly Agree, "A" if 
you Agree, "U" if you are Undecided, "D" if you Disagree, or "SD" if you strongly Disagree 
with the statement. Please make sure to print your name and clock number in the space 
provided. 



1. 


My communications skills have improved as a SA 
result of my involvement in the New Horizons program. 


A 


U 


D 


SD 


2. 


My productivity has increased as a result of my 
invnlvpmpnt in thp Npw Hnn/on^ program 


SA 


A 


U 


D 


SD 


3. 


My attendance record has improved as a result 
nf mv involvpmpnt in the New Horizons Dro&ram 


SA 


A 


u 


D 


SD 


4. 


My self esteem has improved after taking part 

in thf^ Mp\i/ Hon 7nnc PrnoTJim 
Hi mc l^cw jn.vjiiz.vjuo rivj^idiu. 


SA 


A 


u 


D 


SD 


5. 


I enjoyed the training classes. 


SA 


A 


u 


D 


SD 


6. 


1 Iparnpd a lot from mv instructor 


SA 


A 


u 


D 


SD 


7. 


I reached the goals I set in my IEP. 


SA 


A 


u 


D 


SD 


o . 


My New Horizons training has given me the 
self-confidence to take tests for jobs that pay 
more. 


SA 


A 


u 


D 


SD 


9. 


I liked the awards given when I completed 
my training. 


SA 


A 


u 


D 


SD 


10. 


My New Horizons training has helped make 
me a better worker. 


SA 


A 


u 


D 


SD 


11. 


Having my training during working hours 
was great for me. Better than if the training 
had been after work on my own time. 


SA 


A 


u 


D 


SD 


12. 


I have a better work environment because 


SA 


A 


u 


D 


SD 



of my New Horizons training. 
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Analysis Matrix of Project Objectives 



No. 


Objective 


Met 9 


Source of Data 


1.1 


To identify and select instructors who have a 
successful background in Adult Education and 
job-related workplace training. 


MET 




1.2 


To conduct training and orientation sessions for 
all instructors, which will: review effective 
instructional techniques, adult learning theories, 
program goals and objectives, course goals and 
objectives, Individualized Educational Plans and 
program policies and procedures. 






1.3 


To conduct, throughout the project, ongoing 
assessments and evaluations of all instructors. 






2.1 


To conduct and complete comprehensive needs 
assessments to verify the workers who require 
basic skills training and to verify the specific 
job related basic skills required for each 
participant to most effectively perform his/her 
job and become more productive. 




• 


2.2 


To develop competencies (based on the needs 
assessments) for each identified skill. 






3.1 


To write measurable performance objectives. 






3.2 


To develop course measurable objectives . 






4.1 


To develop course objectives and course 
outlines. 






4.2 


To design course content to reflect the 
competencies and skills taught. 






4.3 


To develop job related curriculum materials, 
based on the Literacy Task Analysis, that 
enhance adult learning and which are designed 
to meet the stated competencies and 
performance objectives. 







Page 33 

* r* 

4b 



5.1 


To develop a written Individualized Educational 
Plan for each participant. The Individualized 
Educational Plans will be negotiated and 
ueveiopeu oy me panicipanis anu ine 
instructors. 






6.1 


To conduct basic skills training for at least 484 
oeorgeiown oieei employees uunng ine projeci 
period. 






6.2 


To offer classes and instruction in: Math and 
ixeauing, writing, rrooiem solving anu 
Decision Making , and Time Management . 






6.3 


To offer a program option of one-on-one 
luionng tot xnose wun no, or exirerneiy iuw, 
reading skills. 






6.4 


To provide high school completion/GED 
preparation classes and training to all who 
desire, or need, to teach this educational level. 






6.5 


To provide class schedules and class sizes 
which will meet the needs of all participants 
and enhance learning. Class sizes will be 12-15 
students per class. Classes will be offered 
Monday through Friday. Four class sessions 
will be offered each day: 7:00 a.m. - 9:00 a.m., 
9:00 a.m. - 1 1 :00, 1 :00 p.m. - 3:00 p.m. and 
3:00 p.m. - 5:00 p.m. Two classes of 12-15 
students per class will be held during each two- 
hour session. Because workers work on a 
regular schedule of rotating shifts, these 
sessions will accommodate the needs of all 
participants. 
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6.6 


To provide an opportunity for every participant 
to achieve his/her basic skills 
improvement goals. Each participant will be 
scheduled two hours per day (ten hours per 
week) until the competency level objectives 
defined in the Individualized Educational Plan 
have been attained. The hours of training 
completed by each worker may range from 30 
hours to 100+ hours depending upon the basic 
skills competencies needed. As participants 
attain their competencies, others will be 
scheduled for training. 






7.1 


To conduct a comprehensive evaluation of 
trainee outcomes as well as an evaluation of the 
training program. 






8.1 


To develop and implement a multifaceted media 

campaign to promote the project and 

inform the community about the benefits of the 

program. 




• 


8.2 


The Project Director will generate a quarterly 
report which will be submitted to key project 
personnel. Appropriate information included in 
the report will be made available for press 
releases and public service announcements. 






8.3 


Upon completion of the first year, at least two 
articles will be submitted to journals in 
the fields of Adult Education and Industrial 
Management. Articles will describe the project 
and include information beneficial to other 
corporations and colleges considering similar 
relationships. 
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8.4 



Upon completion of the project, the Director 
will: (1) submit a Final Report to the US.D. of 
E., (2) hold a news conference summarizing the 
project and its benefits, (3) submit at least three 
articles to three journals with national 
circulations, (4) hold a conference 
open to the public, other agencies, corporations 
and institutions, and (5) disseminate 
a summary of the project and copies of 
published articles to the South Carolina Work 
Force Initiative Program Specialists. 
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Full Data Base Development Format 



Purpose: This file should be created on WordPerfect 5.1 and will be used to determine the 
effectiveness of the project. 

x x x x x 

abcdeOOOOOOOOOOl 1 1111111 12222222222333333333344444444445555555555666666666677777777778 
abcdeO 12345678901 234567890123456789012345678901234567890123456789012345678901234567890 



Column 



Data 



abcde Student Clock Number 

0-10 Student Name (Last Name and Initial) 

11 Sex - "1" = Male, "2" = Female 

12 Race - "1" = White. "2" = Black, "3" = American Indian 
"4" = Asian, "5" = Other 

13-14 Highest Grade in School Completed 

(Use the numbers 1-12 and 13 for Beyond High School) 
15-16 Years with the Company 

17 SKIP A COLUMN (BLANK) 

18 Student Shift "1" = First Shift, "2" = Second Shift, 
"3" = Third Shift 

19-20 Job Code (Based on the assigned job code used by GTS) 



21 


Problem Solving - 


"0" 


= No Participation 






"1" 


= Participation 


22 


Reading 


"0" 


= No Participation 






"1" 


= Participation 


23 


Math 


"0" 


= No Participation 






"1" 


= Participation 


24 


Writing 


"0" 


= No Participation 






"1 " 


= Participation 


25 


Time Mgt. 


"0" 


= No Participation 






"1" 


= Participation 


26 


Zapp 


"0" 


= No Participation 






"1" 


= Participation 


27 


TBA 


"0" 


= No Participation 






"1" 


= Participation 


28 


SKIP A COLUMN (BLANK) 




29-31 


Math Pre Test Total Score 






32-34 


Math Post Test Total Score 






35 


SKIP A COLUMN (BLANK) 
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50 



JO-JO 


ixeaciing rre icsi luiai oluic 


1 Q A 1 


ixcaaing rOSl 1 CM lUlal oLUlc 


HZ. 


^KTP A POI UMN (BLANKS 

o r\. 1 1 r\ LULU ivi i ^ ^uljAi^iiv/ 


43-44 


Writing Pre Test Total Score 


45-46 


Writing Post Test Total Score 


47 


SKIP A COLUMN (BLANK) 


48-59 


Item Responses to the Student Survey Form 




(Form to Be Developed during September Visit) 



REMAINING COLUMNS (60-80) TO BE USED AS NEEDED 
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NEW HORIZONS QUARTERLY SURVEY 
REVISED VERSION 

Of the employees from your department who have participated in the New Horizons 

training: 

How many have improved their communications skills 0 

Which job behaviors indicate these improvements 0 

Can you give some examples 9 



How many have increased their productivity 0 

Which job behaviors indicate these improvements 0 



Can you give some examples 0 

How many have improved attendance records 0 

Can you give some examples 

How many do you feel have improved self esteem after participating in the New Horizons 
Program 0 

Which job or personal behaviors indicate these improvements 0 



Can you give some examples? 
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APPENDIX B: "CHARTS" 

New Horizons Participant Survey 
Pre and Post Test Comparisons 
GSC Key Indicator Data Charts 
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New Horizons Participant Survey - Q#2 

Increased Productivity? 




1 



New Horizons Participant Survey - Q#5 

Enjoyed Classes? 




New Horizons Participant Survey - Q#6 

I Learned from Instructor? 




New Horizons Participant Survey - Q#11 

Training During Work Better? 



100 -r 



90 - 




Reaction 

□ Agree = 80% □ Undecided = 8% E3 Disagree = 12% 
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New Horizons Participant Survey - Q#12 

Better Work Environment? 




INCREASE = 3% from 1991 to 1993 




i 
I 



5 T- 



REDUCTION OF DELAY = 11% From 1991 to 1993 



GSC KEY INDICATORS 

Melt Shop % Maintenance Delay 



4 - 



I 
I 



I 



9 

I ERIC 



P 

e 
* 
c 
e 
n 



M 
a 
i 

n 

D 
e 
I 

a 



3 - 



2 - 



1 




□ 1991 = 3.63 



Years 
□ 1992 = 3.37 



□ 1993 = 3.22 
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REDUCTION OF CLAIMS FOR QUALITY = 72% From 1991 to 1993 



GSC KEY INDICATORS 

Cu3tomer Claims for Quality in Dollars 



1 



1 



ERIC 



900 -r 



800 - 



700 - 



600 - 



T 

h 
o 
u 
s 
a 
n 
d 
s 



f 
$ 
s 



500 X 



400 -f- 



300 - 



200 - 



100 - 




□ 1991 = $856,269 



Years 
□ 1992 = 3581,604 
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a 1993 = $242,283 



I 



REDUCTION OF COMPENSIBLES = 11% From 1991 to 1993 



GSC KEY INDICATORS 

Compensibles 



I 

I 



i 



1 
I 



9 

ERIC 



140 



120 - 



100 



N 
u 
m 
b 
e 
r 



80 



60 - 



40 4- 



20 - 




□ 1991 = 134 



Years 
□ 1992 = 106 
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1993 = 120 



